Technical Field
The present invention relates to enterprise networks and, more particularly, to a system and method for real-time detection of abnormal network connections in streaming data.
Description of the Related Art
A typical enterprise network contains hundreds, even thousands of hosts, e.g., servers, desktops, laptops, etc. A single host may generate hundreds of network connections in a second. The total data volume in a middle-sized enterprise network can easily reach terabyte scale in a few hours. Enterprise networks have huge complexity in network structure and the contained entities, and both are evolving over time. The system needs to track the changes and always maintain the model. In security-oriented missions (e.g., intrusion detection), the response time is a critical issue. Many security actions have to be implemented in a short period of time to stop the damage. Thus, the system is required to process the data and detect abnormal connections in real time.
The training data are hard to get in the real applications. It is costly and error-prone to manually label the large dataset of network connections. In addition, the users usually have no knowledge about the abnormal connection patterns and can hardly define any useful models in advance. However, the end users are not satisfied by only being informed of the abnormal network connections. The users also want to know the reason for the abnormal connection in order to implement actions to solve the issues. For example, if the system reports that the connection is abnormal because an unseen process connects via a port which is used by an ftp protocol, then the users may investigate the ftp server to, e.g., cut the connection.